{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WfGpInivS0fG"
   },
   "source": [
    "<h2 align=\"center\">点击下列图标在线运行HanLP</h2>\n",
    "<div align=\"center\">\n",
    "\t<a href=\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_mtl.ipynb\" target=\"_blank\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\t<a href=\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsdp_mtl.ipynb\" target=\"_blank\"><img src=\"https://mybinder.org/badge_logo.svg\" alt=\"Open In Binder\"/></a>\n",
    "</div>\n",
    "\n",
    "## 安装"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install hanlp -U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载模型\n",
    "HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "IYwV-UkNNzFp",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\n",
       " 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\n",
       " 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\n",
       " 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\n",
       " 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\n",
       " 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\n",
       " 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\n",
       " 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import hanlp\n",
    "hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Uf_u7ddMhUt",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "pp-1KqEOOJ4t",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "elA_UyssOut_"
   },
   "source": [
    "## 语义依存分析\n",
    "任务越少，速度越快。如指定仅执行语义依存分析："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "BqEmDMGGOtk3",
    "outputId": "2a0d392f-b99a-4a18-fc7f-754e2abe2e34"
   },
   "outputs": [],
   "source": [
    "doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='sdp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"tok/fine\": [\n",
      "    \"2021年\",\n",
      "    \"HanLPv2.1\",\n",
      "    \"为\",\n",
      "    \"生产\",\n",
      "    \"环境\",\n",
      "    \"带来\",\n",
      "    \"次\",\n",
      "    \"世代\",\n",
      "    \"最\",\n",
      "    \"先进\",\n",
      "    \"的\",\n",
      "    \"多\",\n",
      "    \"语种\",\n",
      "    \"NLP\",\n",
      "    \"技术\",\n",
      "    \"。\"\n",
      "  ],\n",
      "  \"sdp\": [\n",
      "    [[6, \"Time\"]],\n",
      "    [[6, \"Exp\"]],\n",
      "    [[5, \"mPrep\"]],\n",
      "    [[5, \"Desc\"]],\n",
      "    [[6, \"Datv\"]],\n",
      "    [[13, \"dDesc\"]],\n",
      "    [[0, \"Root\"], [8, \"Desc\"], [13, \"Desc\"]],\n",
      "    [[15, \"Time\"]],\n",
      "    [[10, \"mDegr\"]],\n",
      "    [[15, \"Desc\"]],\n",
      "    [[10, \"mAux\"]],\n",
      "    [[8, \"Quan\"], [13, \"Quan\"]],\n",
      "    [[15, \"Desc\"]],\n",
      "    [[15, \"Nmod\"]],\n",
      "    [[6, \"Pat\"]],\n",
      "    [[6, \"mPunc\"]]\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`doc['sdp']`字段代表语义依存图的数组格式，数组中第`i`个子数组代表第`i`个单词的语义依存关系，子数组中每个二元组的格式为`[中心词的下标, 与中心词的语义依存关系]`。每个单词的语义依存关系可能有零个、一个或多个（任意数量）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "转换为[CoNLLSentence](https://hanlp.hankcs.com/docs/api/common/conll.html#hanlp_common.conll.CoNLLSentence)格式更容易观察："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\t2021年\t_\t_\t_\t_\t_\t_\t6:Time\t_\n",
      "2\tHanLPv2.1\t_\t_\t_\t_\t_\t_\t6:Exp\t_\n",
      "3\t为\t_\t_\t_\t_\t_\t_\t5:mPrep\t_\n",
      "4\t生产\t_\t_\t_\t_\t_\t_\t5:Desc\t_\n",
      "5\t环境\t_\t_\t_\t_\t_\t_\t6:Datv\t_\n",
      "6\t带来\t_\t_\t_\t_\t_\t_\t13:dDesc\t_\n",
      "7\t次\t_\t_\t_\t_\t_\t_\t0:Root|8:Desc|13:Desc\t_\n",
      "8\t世代\t_\t_\t_\t_\t_\t_\t15:Time\t_\n",
      "9\t最\t_\t_\t_\t_\t_\t_\t10:mDegr\t_\n",
      "10\t先进\t_\t_\t_\t_\t_\t_\t15:Desc\t_\n",
      "11\t的\t_\t_\t_\t_\t_\t_\t10:mAux\t_\n",
      "12\t多\t_\t_\t_\t_\t_\t_\t8:Quan|13:Quan\t_\n",
      "13\t语种\t_\t_\t_\t_\t_\t_\t15:Desc\t_\n",
      "14\tNLP\t_\t_\t_\t_\t_\t_\t15:Nmod\t_\n",
      "15\t技术\t_\t_\t_\t_\t_\t_\t6:Pat\t_\n",
      "16\t。\t_\t_\t_\t_\t_\t_\t6:mPunc\t_\n"
     ]
    }
   ],
   "source": [
    "print(doc.to_conll())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XOsWkOqQfzlr"
   },
   "source": [
    "为已分词的句子执行语义依存分析："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "id": "bLZSTbv_f3OA",
    "outputId": "111c0be9-bac6-4eee-d5bd-a972ffc34844"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\tHanLP\t_\t_\t_\t_\t_\t_\t5:Exp\t_\n",
      "2\t为\t_\t_\t_\t_\t_\t_\t4:mPrep\t_\n",
      "3\t生产\t_\t_\t_\t_\t_\t_\t4:Desc\t_\n",
      "4\t环境\t_\t_\t_\t_\t_\t_\t5:Datv\t_\n",
      "5\t带来\t_\t_\t_\t_\t_\t_\t0:Root\t_\n",
      "6\t次世代\t_\t_\t_\t_\t_\t_\t12:Time\t_\n",
      "7\t最\t_\t_\t_\t_\t_\t_\t8:mDegr\t_\n",
      "8\t先进\t_\t_\t_\t_\t_\t_\t12:Desc\t_\n",
      "9\t的\t_\t_\t_\t_\t_\t_\t8:mAux\t_\n",
      "10\t多语种\t_\t_\t_\t_\t_\t_\t12:Desc\t_\n",
      "11\tNLP\t_\t_\t_\t_\t_\t_\t12:Nmod\t_\n",
      "12\t技术\t_\t_\t_\t_\t_\t_\t5:Pat\t_\n",
      "13\t。\t_\t_\t_\t_\t_\t_\t5:mPunc\t_\n",
      "\n",
      "1\t我\t_\t_\t_\t_\t_\t_\t3:Poss\t_\n",
      "2\t的\t_\t_\t_\t_\t_\t_\t1:mAux\t_\n",
      "3\t希望\t_\t_\t_\t_\t_\t_\t4:Exp\t_\n",
      "4\t是\t_\t_\t_\t_\t_\t_\t11:mMod\t_\n",
      "5\t希望\t_\t_\t_\t_\t_\t_\t4:dClas\t_\n",
      "6\t张晚霞\t_\t_\t_\t_\t_\t_\t8:Poss\t_\n",
      "7\t的\t_\t_\t_\t_\t_\t_\t6:mAux\t_\n",
      "8\t背影\t_\t_\t_\t_\t_\t_\t11:Pat\t_\n",
      "9\t被\t_\t_\t_\t_\t_\t_\t10:mPrep\t_\n",
      "10\t晚霞\t_\t_\t_\t_\t_\t_\t11:Exp\t_\n",
      "11\t映红\t_\t_\t_\t_\t_\t_\t5:dCont\t_\n",
      "12\t。\t_\t_\t_\t_\t_\t_\t4:mPunc\t_\n"
     ]
    }
   ],
   "source": [
    "print(HanLP([\n",
    "    [\"HanLP\", \"为\", \"生产\", \"环境\", \"带来\", \"次世代\", \"最\", \"先进\", \"的\", \"多语种\", \"NLP\", \"技术\", \"。\"],\n",
    "    [\"我\", \"的\", \"希望\", \"是\", \"希望\", \"张晚霞\", \"的\", \"背影\", \"被\", \"晚霞\", \"映红\", \"。\"]\n",
    "  ], tasks='sdp', skip_tasks='tok*').to_conll())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 注意\n",
    "Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "sdp_mtl.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
